From NvmExpressDxe\NvmExpressPassthru.c, function NvmExpressPassThru:

Before this 'while' loop, the code has initiated an NVMe transaction via a submission queue. The 'while' loop is looking for a state change of 'Cq->Pt' indicating that the transaction is complete.

'Cq' is a pointer to 'NVME_CQ' which is a structure that is updated by the Nvme controller. This structure should have the 'volatile' keyword to ensure that the memory is re-read within the 'while' loop. Without 'volatile', the compiler is only reading the 'Cq->Pt' once outside of the 'while' loop and then doing register compares within the 'while' loop.

It has been observed (on AARCH64 target hardware) that in some cases, 'Cq->Pt' has not yet been updated by the NVMe controller by the time the code reaches the 'while' loop. In those cases, the 'while' loop will only terminate due to timeout because the external memory is no longer being read within the 'while' loop.

A similar code construct is in NvmExpressPei\NvmExpressPassThru.c and there may be other areas where 'volatile' is missing.

Using 'volatile' with declarations of pointer to NVME_CQ results in code that re-reads the memory within the 'while' loop ensuring that the updated state of 'Cq->Pt' is detected.

The problem is intermittent and may depend on underlying hardware and toolchain optimization flags.

Issue any commands/procedures to read from an NVMe drive. Observe NVMe command timeout errors through logs and addition of debug prints in NvmExpressPassThru.c. Code inspection and analysis of assembly listing of execution binaries with and without 'volatile' keyword.

Medium

Someone else needs to fix it

Maintainer feedback requested

There may be other modules besides NvmExpressDei and NvmExpressPei that are missing the 'volatile' keyword.